Immersive display apparatus and method for creation of peripheral view corresponding to input video

ABSTRACT

The present disclosure relates to an immersive display apparatus and method for creation of a peripheral view image corresponding to an input video, the method comprising a pre-processing step of obtaining scene-space information at a main-view video signal corresponding to a first area, a pre-warping step of performing first warping to at least one neighborhood frame corresponding to a target frame included in the pro-processed video signal and determining an outlier from the result of the first warping, a sampling step of sampling at least one neighborhood frame to be used for extrapolation from the result of the first warping, a warping step of performing second warping to the sampled frame except for the outlier to generate a peripheral view image signal corresponding to a second area around the first area, and a blending step of blending the peripheral view image signal to the main-view video signal.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority of Korean Patent Application No.10-2017-0041918, filed on Mar. 31, 2017, in the KIPO (KoreanIntellectual Property Office), the disclosure of which is incorporatedherein entirely by reference.

BACKGROUND OF THE INVENTION Field of the Invention

This disclosure is directed to providing an immersive display apparatusand method for creation of a peripheral view image from a video signalwith a narrow view angle in order to generate an image with a wide viewangle by processing a video signal with a narrow view angle.

Description of the Related Art

An immersive display apparatus provides a wider view angle to a user incomparison to an existing display apparatus so that the user may beimmersed in a video more deeply.

For example, in an existing immersive display apparatus, a plurality ofcameras for providing a video of a wider view angle are used tophotograph a video at various angles, and then the photographed videosare combined.

However, using a plurality of cameras increases costs, and the amount ofdata to be process may increase excessively.

In another example, in an existing immersive display apparatus, a camerahaving a large photographing angle may be used to make an image.

In this case, the image may have lowered resolution.

SUMMARY OF THE INVENTION

This disclosure is directed to providing an immersive display apparatusand method for creation of a peripheral view image from a video signalwith a narrow view angle in order to generate an image with a wide viewangle by processing a video signal with a narrow view angle

In one general aspect of the present disclosure, there is provided animmersive display method for creation of a peripheral view imagecorresponding to an input video, comprising: a pre-processing step ofobtaining scene-space information at a main-view video signalcorresponding to a first area; a pre-warping step of performing firstwarping to at least one neighborhood frame corresponding to a targetframe included in the pro-processed video signal and determining anoutlier from the result of the first warping; a sampling step ofsampling at least one neighborhood frame to be used for extrapolationfrom the result of the first warping; a warping step of performingsecond warping to the sampled frame except for the outlier to generate aperipheral view image signal corresponding to a second area around thefirst area; and a blending step of blending the peripheral view imagesignal to the main-view video signal.

In addition, the scene-space information may include scene pointinformation and camera parameter information.

In addition, the neighborhood frame and the target frame may be includedin the same shot.

In addition, frames sampled corresponding to all target frames of aframe group included in the same shot may be identical to each other.

In addition, each frame included in the frame group may be a targetframe.

In addition, assuming that a scene point on the neighborhood framecorresponding to a first scene point on the target frame is called asecond scene point and a scene point obtained by performing the firstwarping to the second scene point is a second warping scene point, inthe pre-warping step, the second scene point may be set as an outlierwhen a distance between the first scene point and the second warpingscene point is greater than a preset reference distance.

In addition, the sampled frame may include a portion not overlappingwith the target frame.

In addition, in the blending step, the peripheral view image signal maybe blended to the main-view video signal in the order closer to themain-view video image with respect to the target frame.

In addition, a resolution corresponding to the main-view video signalmay be identical to a resolution corresponding to the peripheral viewimage signal.

In another aspect of the present disclosure, there is provided animmersive display apparatus for creation of a peripheral view imagecorresponding to an input video, comprising: a display part configuredto display an image; a pre-processing part configured to obtainscene-space information from a main-view video signal corresponding to afirst area of the display part; a pre-warping part configured to performfirst warping to at least one neighborhood frame corresponding to atarget frame included in the video signal pro-processed by thepre-processing part and determine an outlier from the result of thefirst warping; a sampling part configured to sample at least oneneighborhood frame to be used for extrapolation from the result of thefirst warping; a warping part configured to perform second warping tothe sampled frame except for the outlier to generate a peripheral viewimage signal corresponding to a second area around the first area of thedisplay part; and a blending part configured to blend the peripheralview image signal to the main-view video signal.

In addition, the scene-space information may include scene pointinformation and camera parameter information.

In addition, the neighborhood frame and the target frame may be includedin the same shot.

In addition, frames sampled corresponding to all target frames of aframe group, which has a plurality of frames, included in the same shotmay be identical to each other.

In addition, each frame included in the frame group may be a targetframe.

In addition, assuming that a scene point on the neighborhood framecorresponding to a first scene point on the target frame is called asecond scene point and a scene point obtained by performing the firstwarping to the second scene point is a second warping scene point, thepre-warping part may set the second scene point as an outlier when adistance between the first scene point and the second warping scenepoint is greater than a preset reference distance.

In addition, the sampled frame may include a portion not overlappingwith the target frame.

In addition, the blending part may blend the peripheral view imagesignal to the main-view video signal in the order closer to themain-view video image with respect to the target frame.

In addition, a resolution corresponding to the main-view video signalmay be identical to a resolution corresponding to the peripheral viewimage signal.

In another aspect of the present disclosure, there is provided animmersive display apparatus for creation of a peripheral view imagecorresponding to an input video, comprising: a first display partincluding a first display area; a second display part including a seconddisplay area; a pre-processing part configured to obtain scene-spaceinformation from a main-view video signal corresponding to the firstdisplay part; a pre-warping part configured to perform first warping toat least one neighborhood frame corresponding to a target frame includedin the video signal pro-processed by the pre-processing part anddetermine an outlier from the result of the first warping; a samplingpart configured to sample at least one neighborhood frame to be used forextrapolation from the result of the first warping; a warping partconfigured to perform second warping to the sampled frame except for theoutlier to generate a peripheral view image signal corresponding to thesecond display part; and a blending part configured to blend theperipheral view image signal to the main-view video signal.

In addition, the scene-space information may include scene pointinformation and camera parameter information.

The immersive display apparatus and method for creation of a peripheralview image according to the present disclosure generates a peripheralview image corresponding to a target frame by using video information ofa peripheral frame adjacent to the target frame of an input videosignal, thereby preventing deterioration of resolution of the peripheralview image.

In addition, the immersive display apparatus and method for creation ofa peripheral view image according to the present disclosure may reducetime required for processing data.

Moreover, the immersive display apparatus and method for creation of aperipheral view image according to the present disclosure may suppressdistortion of the peripheral view image.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages will become more apparent tothose of ordinary skill in the art by describing in detail exemplaryembodiments with reference to the attached drawings, in which:

FIGS. 1 to 3 are diagrams for illustrating an immersive displayapparatus for creation of a peripheral view image according to thepresent disclosure.

FIGS. 4 to 18 are diagrams for illustrating an immersive display methodfor creation of a peripheral view image according to the presentdisclosure.

In the following description, the same or similar elements are labeledwith the same or similar reference numbers.

DETAILED DESCRIPTION

The present invention now will be described more fully hereinafter withreference to the accompanying drawings, in which embodiments of theinvention are shown. This invention may, however, be embodied in manydifferent forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the invention to those skilled in the art.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “includes”,“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof. In addition, a term such asa “unit”, a “module”, a “block” or like, when used in the specification,represents a unit that processes at least one function or operation, andthe unit or the like may be implemented by hardware or software or acombination of hardware and software.

Reference herein to a layer formed “on” a substrate or other layerrefers to a layer formed directly on top of the substrate or other layeror to an intermediate layer or intermediate layers formed on thesubstrate or other layer. It will also be understood by those skilled inthe art that structures or shapes that are “adjacent” to otherstructures or shapes may have portions that overlap or are disposedbelow the adjacent features.

In this specification, the relative terms, such as “below”, “above”,“upper”, “lower”, “horizontal”, and “vertical”, may be used to describethe relationship of one component, layer, or region to anothercomponent, layer, or region, as shown in the accompanying drawings. Itis to be understood that these terms are intended to encompass not onlythe directions indicated in the figures, but also the other directionsof the elements.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Preferred embodiments will now be described more fully hereinafter withreference to the accompanying drawings. However, they may be embodied indifferent forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the disclosure to those skilled in the art.

FIGS. 1 to 3 are diagrams for illustrating an immersive displayapparatus for creation of a peripheral view image according to thepresent disclosure.

Referring to FIG. 1, an immersive display apparatus according to thepresent disclosure may include a display device 10 having a plurality ofdisplay parts 11, 12, 13 and a driver 14 for processing an input videosignal through a predetermined procedure and supplying the processedsignal to the display device 10.

For example, the display device 10 may include a first display part 11having a first display area and second display parts 12, 13 having asecond display area.

A main-view video image may be displayed in the first display area ofthe first display part 11, and a peripheral view image may be displayedin the second display area of the second display parts 12 and 13.

Considering this, the first display part 11 may be referred to as a maindisplay part, and the second display parts 12 and 13 may be referred toas a peripheral display part.

A video signal corresponding to the first display part 11 may bereferred to as a main-view video signal, and a video signalcorresponding to the second display part may be referred to as aperipheral view image signal.

The first display part 11 and the second display parts 12 and 13 may bearranged to surround the user. In this case, the user may feel astronger immersion feeling from the main-view video image displayed onthe first display part 11 and the peripheral view image displayed on thesecond display parts 12 and 13.

A plurality of second display parts 12, 13 may be arranged around thefirst display part 11. For example, it is possible that the second-1display part 12 is disposed at the right side of the first display part11 and the second-2 display part 13 is disposed at the left side of thefirst display part 11.

Even though FIG. 1 illustrates the case where two auxiliary displayparts are arranged around the first display part 11, the presentdisclosure may not be limited thereto. For example, it is possible thatarrange auxiliary display parts are arranged at top, bottom, left, andright sides of the first display part 11, respectively. Alternatively,it is also possible that either the second-1 display part 12 or thesecond-2 display part 13 is excluded.

The driver 14 may process the input video signal to generate aperipheral view image to be displayed on the second display parts 12 and13.

As shown in FIG. 2, the driver 14 may include a pre-processing part 100,a pre-warping part 110, a sampling part 120, a warping part 130, ablending part 140, and a controller 150.

The pre-processing part 100 may process the input video signal to obtainscene-space information.

The video signal processed by the pre-processing part 100 may be a videosignal corresponding to the first area of the display part 10, namelythe first display part 11. If the driver 14 does not generate aperipheral view image, an image is displayed in the first display part11, but no image may be displayed in the second display parts 12 and 13,corresponding to the input video signal.

The scene-space information obtained by the pre-processing part 100 isinformation necessary for warping a video signal, and may include scenepoint information and camera parameter information. Here, the scenepoint information may be 3D scene point information.

The scene point information and the camera parameter information may beobtained through the structure-from-motion (SFM) technique. Thestructure-from-motion (SFM) technique is already known in the art andthus not described in detail here.

Here, the scene point information may be information about a portionhaving predetermined image data on an image.

The pre-warping part 110 may perform first warping to at least oneneighborhood frame corresponding to a target frame included in thepro-processed video signal. Here, performing the first warping to theneighborhood frame may be regarded as that the video signalcorresponding to the neighborhood frame is primarily warped.

Warping may mean that data about an image is transformed by means ofshifting or the like based on information such as parallax of the image.

The pre-warping part 110 may determine an outlier from the result of thefirst warping.

The method of determining the neighborhood frame and the method ofdetermining the outlier will be explained more clearly in the followingdescription.

The sampling part 120 may sample at least one neighborhood frame to beused for extrapolation from the result of the first warping by thepre-warping part 110. In other words, the sampling part 120 may selectat least one neighborhood frame suitable for generating a peripheralview image.

The warping part 130 may perform second warping while excluding theoutlier for the sampled frame by the sampling part 120. Here, performingthe second warping to the sampled frame may be regarded as that a videosignal corresponding to the neighborhood frame selected by the samplingpart 120 is secondarily warped.

In addition, the warping part 130 may generate a peripheral view imagesignal corresponding to a second area around the first area of thedisplay part 10, namely the second display parts 12 and 13, from theresult of the second warping.

In other words, the warping part 130 may generate a peripheral viewimage by warping the sampled frame except for the outlier.

The warping part 130 may obtain information about the outlier from thepre-warping part 110 to perform the second warping.

The blending part 140 may blend the peripheral view image signalgenerated by the warping part 130 to the main-view video signal.

The display part 10 may display an image corresponding to the videosignal blended by the blending part 140 so that the image may be viewed.

The controller 150 may perform overall control functions for imageprocessing and image displaying.

If the pre-processing part 100, the pre-warping part 110, the samplingpart 120, the warping part 130 and the blending part 140 performdetermination and control functions independently, the controller 150may be excluded.

The pre-warping part 110 and the warping part 130 may be integrated intoa single part in that they perform warping.

Meanwhile, even though it has been described above that the main-viewvideo image is displayed on the first display part 11 and the peripheralview image is displayed on the second display parts 12 and 13 differentfrom the first display part 11, the present disclosure may be notlimited thereto.

For example, as shown in FIG. 3, a display part 10 a may include a firstarea 11 a included in one screen and second areas 12 a and 13 asurrounding the first area 11 a.

In this case, the main-view video image may be displayed in the firstarea 11 a of the display part 10 a and the peripheral view image may bedisplayed in the second areas 12 a, 13 a of the display part 10 a.

An immersive display method according to the present disclosure, whichwill be described below, may be applied to various types of displaydevices.

FIGS. 4 to 18 are diagrams for illustrating an immersive display methodfor creation of a peripheral view image according to the presentdisclosure. In the following disclosure, any portion already explainedabove may not be described again.

Referring to FIG. 4, in the immersive display method according to thepresent disclosure, if a main-view video signal corresponding to thefirst area is input (S100), the input video signal may be pre-processed(S200). By means of this pre-processing process, scene-space informationcorresponding to the main-view video signal may be obtained.

For example, in the pre-processing step (S200), information about ascene point and information about a camera parameter for each frameincluded in the video signal may be obtained using thestructure-from-motion (SFM) technique.

After that, the neighborhood frame NF corresponding to the target frameTF may be identified (S210). In detail, the neighborhood frame NFcorresponding to the target frame TF may be identified based on thescene-space information obtained in the pre-processing step (S200).Here, the neighborhood frame NF may be a frame adjacent to the targetframe TF in time.

In addition, the neighborhood frame NF and the target frame TF may beincluded in the same shot (a single shot).

Considering the above, it is possible to set the target frame TF and theneighborhood frame NF corresponding thereto in a frame group (FG) havinga plurality of frames included in the same shot.

For example, as shown in FIG. 5, it may be assumed that a scene changesat a time t1 between a first frame F1 and a second frame F2, and then ascene changes again at a time t2 between an n^(th) frame Fn and ann+1^(th) frame Fn+1.

In this case, it may be regarded that the second frame F2 to the n^(th)frame Fn are photographed corresponding to the same camera shot. Fromthis, the second frame F2 to the n^(th) frame Fn may be regarded as aframe group FG.

It is possible to set all frames except for the target frame TF in theframe group FG as the neighborhood frame NF. For example, in the casedepicted in FIG. 5, if a fourth frame F4 is the target frame TF in theframe group FG, 2^(nd)-3^(rd) and 5^(th)-n^(th) frames may be set asneighborhood frames NF of the fourth frame F4, which is the target frameTF.

Alternatively, it is also possible that a frame selected according to apredetermined criterion among the frames in the frame group FG otherthan the target frame TF is set as the neighborhood frame NF.

As described above, in the case where the neighborhood frame NFcorresponding to the target frame TF is set based on a screen change, itis possible to suppress an excessive increase in the amount of data tobe processed.

In addition, if the neighborhood frame NF is set corresponding to targetframe TF according to the above method, it is possible to sufficientlyensure similarity of image data between the target frame TF and theneighborhood frame NF. In this case, the visual and spatial coherence ofthe main-view video image and the peripheral view image may be improved.

Each frame included in the frame group FG may be a target frame.

The camera parameter corresponding to at least one neighborhood frame NFmay be the same as the camera parameter corresponding to the targetframe TF.

After identifying the neighborhood frame NF corresponding to the targetframe TF, the first warping (S300) may be performed to the video signalcorresponding to the neighborhood frame. For example, the first warpingmay be performed by transforming image data of the neighborhood frame NFto match the viewpoint of the neighborhood frame NF to the viewpoint ofthe target frame TF.

The first warping will be described in more detail as follows withreference to FIG. 6.

The first warping may be performed to determine an outlier to beexcluded in a following warping step (S500) and to provide a criterionfor frame sampling. The outlier may be generated by incompleteness ofthe structure-from-motion (SFM) technique.

The first warping may be achieved by the content-preserving warping(CPW) Technique using a mesh optimization scheme.

In detail, as shown in FIG. 6, the first warping may be performed insuch a way that the sum (Equation 3) of the energy calculated byEquation 1 and the energy calculated by Equation 2 is minimized.

Equations 1, 2, and 3 are described in more detail below.

Equation 1 may mean an energy function for displacement.

In Equation 1, f_(k) ^(M) and f_(k) ^(I) may mean a k^(th) pair ofmatched feature points with a spherical coordinate system

f_(k) ^(M) may correspond to a first video signal converted to thespherical coordinate system, and f_(k) ^(I) may correspond to a secondvideo signal (φ(It)) that is inversely transformed into the sphericalcoordinate system

j may refer to vertex indices (vertex indices), v_(j,k) denoted by anemphasis point may refer to each vertex, and w^(f) _(j,k) may be abarycentric weight of (f_(k) ^(I)).

In Equation 2, R₉₀ may refer to a matrix corresponding to a 90 degreerotation.

v1, v2, v3 denoted by an emphasis point may refer to a warped vertexwith a triangle face f.

Equation 3 may mean an energy function according to Equation 2.

Equation 4 may mean the sum of the energy calculated by Equation 1 andthe energy calculated by Equation 3.

In Equation 4, σ may mean a spatial smoothness weight.

Assuming that the camera parameter of the t^(th) frame is ct, if mnumber of scene points are projected on the t^(th) frame using ct, mnumber of projection points may be generated. Pt={pt¹, pt², . . . ,pt^(m)}

In Equation 1, Ptarget may refer to a projection point on the targetframe TF. In addition, P^(k)target may mean a k^(th) projection point onthe target frame TF.

f(P^(k)t) may be a set of indices for vertices including faces with(P^(k)t).

(W _(t) ^(j)) may be a barycentric weight of (P^(k)t) with respect tothe surrounding vertices.

Here, the projection point Pt may deviate from the field of view (FOV)of the t^(th) frame. In this way, a triangulated grid mesh may begenerated and applied to account for the projected point at a positionbeyond the input frame.

Each frame has an initial grid mesh (V) consisting of {v^(j)}. Here,{v^(j)} may mean a j^(th) grid vertex.

V denoted by an emphasis point may mean a warped mesh consisting of{v^(j)}.

By using Equation 1, the matching of P^(k)target and (P^(k)) may bechecked.

In Equation 2, R₉₀ may refer to a matrix corresponding to a 90 degreerotation.

(v_(t) ^(f1)), (v_(t) ^(f2)) and (v_(t) ^(f3)) denoted by an emphasispoint may refer to a vertex having a triangle face f.

u and v which are coordinate information (local coordinate) may becalculated from the initial grid mesh (V).

By using Equation 2, the maintenance of the initial shape of the initialgrid mesh (V) may be checked.

Equation 3 may refer to the sum of the energy calculated by Equation 1and the energy calculated by Equation 2.

In Equation 3, As may mean a spatial smoothness weight.

The first warping may be performed in such a way that the energycalculated by Equation 3 is minimized.

In addition, the first warping may be performed iteratively for eachtarget frame TF.

The method of using the result of the first warping performed in theabove manner will be described below with reference to FIG. 7.

Referring to FIG. 7, in the pre-warping step S300, the first warping maybe performed to the input image (S310), and the result of the firstwarping may be derived (S320).

A warping error may be determined using the result value for the firstwarping.

For this, it is possible to compare a projection point Ptarget on thetarget frame TF with a projection point Pt to which an emphasis pointcorresponding to the warped t^(th) frame is given (S330).

In addition, a distance between Ptarget and Pt to which the emphasispoint is given may be computed (S340).

After that, if the distance between Ptarget and Pt to which the emphasispoint is given is greater than a predetermined reference distance, theprojection point may be determined as an outlier (S350).

As another method for determining an outlier, assuming that a scenepoint on the neighborhood frame NF corresponding to the first scenepoint on the target frame TF is a second scene point and a scene pointobtained by performing the first warping to the second scene point is asecond warping scene point, the second scene point may be determined asan outlier if the distance between the first scene point and the secondwarping scene point is greater than the preset reference distance.

The outlier determined in this way may be excluded from a followingwarping step. This will be described in more detail later.

Meanwhile, a sampling criterion required for frame sampling may beextracted using the result of the first warping.

For this, a portion (a peripheral view) newly added corresponding to thetarget frame TF may be determined using the result of the first warping(S370).

After that, the sampling criterion may be extracted using informationabout the newly added portion (S380).

After setting the sampling criterion, the neighborhood frame to bewarped may be sampled according to the sampling criterion (S400). Inother words, at least one neighborhood frame to be used forextrapolation may be sampled from the result of the first warping.

The selected neighborhood frame, or the sampled frame, may mean a framecapable of providing effective data for generating a peripheral viewimage corresponding to the target frame TF.

In other words, it is possible to sample a frame that contains datasuitable (good) for generating a peripheral view image.

The sampling criterion is described in more detail below.

The sampled frame may contain a portion that does not overlap with thetarget frame.

For example, it is assumed that an image as shown in FIG. 8A with oneshot. FIG. 8A may be referred to as an image corresponding to one framegroup FG. In FIG. 8, the second, third and fourth frames F2, F3, F4 maybe neighborhood frames NF of the first frame F1, which is the targetframe TF.

As in FIG. 8B, the first frame F1, which is the target frame, maycorrespond to a first portion IP1 of the entire image A.

As in FIG. 8C, the second frame F2 corresponding to the first portionIP1 of the entire image A, identical to the first frame F1, may not besampled.

The second frame F2 may not be sampled because the second frame F2 doesnot contain any information which can be referred to when generating theperipheral view image corresponding to the first frame F1.

It is also possible that the second frame F2 equal to the first frameF1, which is the target frame TF, is set as the sampling frame in orderto prevent errors in the following sampling process.

As shown in FIG. 8D, the third frame F3 may correspond to the secondportion IP2 of the entire image (A). Here, the second portion IP2 may bepartially overlapped with the first portion IP1.

In this case, the third frame F3 may be set as a sampling frame for thefirst frame F1.

As shown in FIG. 8E, the fourth frame F4 may correspond to the thirdportion IP3 of the entire image A. Here, the third portion IP3 may notoverlap with the first portion IP1.

In this case, the fourth frame F4 may be set as a sampling frame for thefirst frame F1.

The fourth frame F4 should be sampled because the fourth frame F4 maycontain information that can be referred to in generating the peripheralview image corresponding to the first frame F1.

Alternatively, it is also possible that at least one neighborhood frameis sampled considering the size of a peripheral view image to be addedto the main-view video image.

For example, it is assumed that the image as in FIG. 9A is photographedwith one shot, and the first frame F1 is the target frame.

As shown in FIGS. 9B and 9C, the first frame F1, which is the targetframe TF, may correspond to the first area 11 a of the display part 10,and the first frame F1 may correspond to the first portion IP1 a of theentire image A.

As shown in FIG. 9D, the second frame F2 may correspond to the secondportion IP2 a of the entire image A. Here, the second portion IP2 a maypartially overlap with the first portion IP1 a, and the second portionIP2 a may partially overlap with the second areas 12 a, 13 a of thedisplay part 10.

In this case, the second frame F2 may be set as a sampling frame for thefirst frame F1.

Meanwhile, as shown in FIG. 9E, the third frame F3 may correspond to thethird portion IP3 a of the entire image A. Here, the third portion IP3 amay not overlap with the first portion IP1 a, and the third portion IP3a may not overlap with the second areas 12 a, 13 a of the display part10.

In this case, the third frame F3 may not be set as the sampling framefor the first frame F1.

The third frame F3 is not sampled because the third frame F3 does notcontain information about the image to be displayed in the second areas12 a, 13 a of the display part 10.

In another case, it is also possible that at least one neighborhoodframe is sampled based on information about the outlier included in theframe.

For example, as shown in FIG. 10, information about the number ofoutliers per frame may be discriminated (S381) from the result of thefirst warping in the sampling reference setting step S380.

After that, it may be determined whether the number of outliers isgreater than a predetermined threshold (S382).

If it is determined that the number of outliers is greater than thethreshold, the frame may be set as an improper frame (S383). Theimproper frame may mean a neighborhood frame that will not be sampled.

Meanwhile, if the number of outliers is smaller than the threshold, thecorresponding frame may be set as an appropriate frame suitable forsampling (S384). The reason for the above sampling is that the number ofoutliers which is relatively small may mean that image distortion is notlarge.

The method of sampling the neighborhood frame NF to be warped in thepresent disclosure may not be limited to the above description. Forexample, it is also possible that a frame to be warped is randomlyselected among a plurality of neighborhood frames NF.

Meanwhile, it is also possible that the frame to be sampled is set inthe same way for each frame group FG.

For example, as shown in FIG. 11, it is assumed that a first frame groupFG1 includes a first frame F1 to an a^(th) frame Fa and a second framegroup FG2 includes an a+1^(th) frames Fa+1 to an n^(th) frame Fn.

In this case, the first sampling frame group SFG1 may be setcorresponding to the first frame group FG1. In other words, all theneighborhood frames sampled corresponding to the respective targetframes included in the first frame group FG1 may be identically set asthe first sampling frame group SFG1.

In this case, the amount of data to be processed may be greatly reduced.

After the frame sampling step (S400), second warping may be performed tothe video signal corresponding to the sampled frame (S500).

The second warping will be described in more detail as follows withreference to FIG. 12.

In the second warping, the sampling frame may be warped except for theoutliers determined in the first warping step (S300).

In detail, as shown in FIG. 12, the second warping may be performed insuch a way that the sum (Equation 7) of the energy calculated byEquation 4, the energy calculated by Equation 5 and the energycalculated by Equation 6 is minimized.

Equations 4, 5, 6, and 7 will be described in more detail below. In thefollowing description, the portions described in Equations 1, 2 and 3may not be described in detail again.

Equation 4 may mean an energy function for inter-frame coherency.

By means of Equation 4, it may be checked whether corresponding scenepoints are matched.

Equation 5 may mean an energy function for preventing deformation of theoriginal information (original content).

Equation 6 may mean an energy function for temporal consistency.

Equation 7 may mean the sum of the energy calculated by Equation 4, theenergy calculated by Equation 5 and the energy calculated by Equation 6.

In Equation 7, λi may mean an inter-frame smoothness weight, λc may meana temporal smoothness weight, and λtf may mean a target frame constraintweight.

The second warping may be performed in such a way that the energycalculated by Equation 7 is minimized.

In addition, the second warping may be iteratively performed for eachtarget frame TF.

A peripheral view image may be generated by means of the second warping.In other words, a peripheral view image signal corresponding to thesecond area around the first area may be generated by performing thesecond warping to the sampled frame, except for the outliers.

As described above, if the second warping is performed so that theenergy calculated by Equation 7 is minimized, it is possible to suppressthe generation of inter-frame and temporal errors (distortion) of theperipheral view image.

For example, if a warping method not conforming to Equation 7 as shownin FIG. 13A is used, inter-frame distortion may be caused in theperipheral view (PV) image around the main-view (MV) image.

Meanwhile, if a warping method conforming to Equation 7 as shown in FIG.13B is used, it is possible to suppress the occurrence of inter-framedistortion in the peripheral view (PV) image.

In the warping process, it is possible to suppress the distortion of theperipheral view image by warping except for the outliers. This will bedescribed below with reference to FIGS. 14 and 15.

FIG. 14A shows outliers and inliers.

The outliers are marked with red dots, and the inliers are marked withgreen dots. Here, the outliers may correspond to a portion having arelatively high possibility of image distortion, and the inliers maycorrespond to a portion having a relatively low possibility ofgenerating image distortion in comparison with the outliers.

If the second warping is performed without excluding the outliers andthen the peripheral view image is generated as a result of the secondwarping, as shown in the purple box of FIG. 14B, distortion may occur inthe peripheral view image.

Meanwhile, if the second warping is performed except for the outliers asshown in FIG. 15A, it is possible to suppress the distortion in theperipheral view image as shown in FIG. 15B.

If FIG. 15A is compared with FIG. 14A, it may be found that the red dotsare disappeared (controlled) on the screen.

After the second warping is performed to the sampled frame except forthe outlier, the peripheral view image generated by the second warpingmay be blended to the main-view video image (S600). In other words, theperipheral view image signal may be blended to the main-view videosignal.

In this blending step, the peripheral view image signal may be blendedto the main-view video signal in the order closer to the main-view videoimage corresponding to the target frame TF.

For example, as shown in FIGS. 16A, 16B and 16C, it is possible that afirst peripheral view (PV1) image closest to the main-view (MV) image isblended, then a second peripheral view (PV2) image is blended to thefirst peripheral view (PV1) image, and then a third peripheral view(PV3) image is blended to the second peripheral view (PV2) image.

Meanwhile, it is also possible that the peripheral view image is blendedto the main-view video image in a plurality of directions.

For example, as shown in FIG. 17, it is possible that the peripheralview image is blended in a left direction of the target framecorresponding to the target frame and also the peripheral view image isblended in a right direction of the target frame.

FIG. 18 shows an example in which a peripheral view image is generatedaccording to the method of the present disclosure, and the generatedperipheral view image is blended to the main view video image anddisplayed.

FIG. 18A may be a main-view video image as a video corresponding to thevideo signal input to the driver 14.

FIG. 18B may be an image obtained by generating a peripheral view image(PVL, PVR) by the driver 14 according to the method of the presentdisclosure and then blending the image with the main-view video image.This image may be referred to as an immersive video.

In the present disclosure, since a video signal corresponding to theperipheral view is generated based on the video signal corresponding tothe main-view, the resolution corresponding to the main-view videosignal and the resolution corresponding to the peripheral view imagesignal may be the same. Thus, the quality of the immersive image may beimproved.

While the present disclosure has been described with reference to theembodiments illustrated in the figures, the embodiments are merelyexamples, and it will be understood by those skilled in the art thatvarious changes in form and other embodiments equivalent thereto can beperformed. Therefore, the technical scope of the disclosure is definedby the technical idea of the appended claims The drawings and theforgoing description gave examples of the present invention. The scopeof the present invention, however, is by no means limited by thesespecific examples. Numerous variations, whether explicitly given in thespecification or not, such as differences in structure, dimension, anduse of material, are possible. The scope of the invention is at least asbroad as given by the following claims.

What is claimed is:
 1. An immersive display method for creation of aperipheral view image corresponding to an input video, comprising: apre-processing step of obtaining scene-space information at a main-viewvideo signal corresponding to a first area; a pre-warping step ofperforming first warping to at least one neighborhood framecorresponding to a target frame included in the pro-processed videosignal and determining an outlier from the result of the first warping;a sampling step of sampling at least one neighborhood frame to be usedfor extrapolation from the result of the first warping; a warping stepof performing second warping to the sampled frame except for the outlierto generate a peripheral view image signal corresponding to a secondarea around the first area; and a blending step of blending theperipheral view image signal to the main-view video signal.
 2. Theimmersive display method of claim 1, wherein the scene-space informationincludes scene point information and camera parameter information. 3.The immersive display method of claim 2, wherein the neighborhood frameand the target frame are included in the same shot.
 4. The immersivedisplay method of claim 3, wherein frames sampled corresponding to alltarget frames of a frame group included in the same shot are identicalto each other.
 5. The immersive display method of claim 4, wherein eachframe included in the frame group is a target frame.
 6. The immersivedisplay method of claim 2, wherein assuming that a scene point on theneighborhood frame corresponding to a first scene point on the targetframe is called a second scene point and a scene point obtained byperforming the first warping to the second scene point is a secondwarping scene point, in the pre-warping step, the second scene point isset as an outlier when a distance between the first scene point and thesecond warping scene point is greater than a preset reference distance.7. The immersive display method of claim 1, wherein the sampled frameincludes a portion not overlapping with the target frame.
 8. Theimmersive display method of claim 1, wherein in the blending step, theperipheral view image signal is blended to the main-view video signal inthe order closer to the main-view video image with respect to the targetframe.
 9. The immersive display method of claim 1, wherein a resolutioncorresponding to the main-view video signal is identical to a resolutioncorresponding to the peripheral view image signal.
 10. An immersivedisplay apparatus for creation of a peripheral view image correspondingto an input video, comprising: a display part configured to display animage; a pre-processing part configured to obtain scene-spaceinformation from a main-view video signal corresponding to a first areaof the display part; a pre-warping part configured to perform firstwarping to at least one neighborhood frame corresponding to a targetframe included in the video signal pro-processed by the pre-processingpart and determine an outlier from the result of the first warping; asampling part configured to sample at least one neighborhood frame to beused for extrapolation from the result of the first warping; a warpingpart configured to perform second warping to the sampled frame exceptfor the outlier to generate a peripheral view image signal correspondingto a second area around the first area of the display part; and ablending part configured to blend the peripheral view image signal tothe main-view video signal.
 11. The immersive display apparatus of claim10, wherein the scene-space information includes scene point informationand camera parameter information.
 12. The immersive display apparatus ofclaim 11, wherein the neighborhood frame and the target frame areincluded in the same shot.
 13. The immersive display apparatus of claim12, wherein frames sampled corresponding to all target frames of a framegroup, which has a plurality of frames, included in the same shot areidentical to each other.
 14. The immersive display apparatus of claim13, wherein each frame included in the frame group is a target frame.15. The immersive display apparatus of claim 11, wherein assuming that ascene point on the neighborhood frame corresponding to a first scenepoint on the target frame is called a second scene point and a scenepoint obtained by performing the first warping to the second scene pointis a second warping scene point, the pre-warping part sets the secondscene point as an outlier when a distance between the first scene pointand the second warping scene point is greater than a preset referencedistance.
 16. The immersive display apparatus of claim 10, wherein thesampled frame includes a portion not overlapping with the target frame.17. The immersive display apparatus of claim 10, wherein the blendingpart blends the peripheral view image signal to the main-view videosignal in the order closer to the main-view video image with respect tothe target frame.
 18. The immersive display apparatus of claim 10,wherein a resolution corresponding to the main-view video signal isidentical to a resolution corresponding to the peripheral view imagesignal.
 19. An immersive display apparatus for creation of a peripheralview image corresponding to an input video, comprising: a first displaypart including a first display area; a second display part including asecond display area; a pre-processing part configured to obtainscene-space information from a main-view video signal corresponding tothe first display part; a pre-warping part configured to perform firstwarping to at least one neighborhood frame corresponding to a targetframe included in the video signal pro-processed by the pre-processingpart and determine an outlier from the result of the first warping; asampling part configured to sample at least one neighborhood frame to beused for extrapolation from the result of the first warping; a warpingpart configured to perform second warping to the sampled frame exceptfor the outlier to generate a peripheral view image signal correspondingto the second display part; and a blending part configured to blend theperipheral view image signal to the main-view video signal.
 20. Theimmersive display apparatus of claim 19, wherein the scene-spaceinformation includes scene point information and camera parameterinformation.